Determining System Behavior Using Event Patterns in Machine Data

ABSTRACT

Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit as a CON of U.S. Ser. No. 15/011,622,filed Jan. 31, 2016, which claims benefit as a CON of U.S. Ser. No.14/611,188, filed Jan. 31, 2015, now U.S. Pat. No. 9,280,591, issuedMar. 8, 2016 which claims benefit as a CON of U.S. Ser. No. 14/170,228,filed Jan. 31, 2014, now U.S. Pat. No. 9,128,916, issued Sep. 9, 2015which claims benefit as a CON of U.S. Ser. No. 13/664,109, filed Oct.30, 2012, now U.S. Pat. No. 8,694,450, issued Apr. 8, 2014, which claimsbenefit as a CON of U.S. Ser. No. 13/099,268, filed May 2, 2011, nowU.S. Pat. No. 8,589,321, issued Nov. 19, 2013, which claims benefit as aCON of U.S. Ser. No. 11/459,632, filed Jul. 24, 2006, now U.S. Pat. No.7,937,344, issued May 3, 2011, which claims benefit of U.S. PRO Ser. No.60/702,496, filed Jul. 25, 2005, the entire contents of theaforementioned are hereby incorporated by reference as if fully setforth herein, under 35 U.S.C. § 120. The applicant(s) hereby rescind anydisclaimer of claim scope in the parent application(s) or theprosecution history thereof and advise the USPTO that the claims in thisapplication may be broader than any claim in the parent application(s).

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates generally to information organization andunderstanding, and more particularly to the organization andunderstanding of machine data.

2. Description of the Related Art

Information systems invariably generate vast amounts and wide varietiesof machine data (e.g., activity logs, configuration files, messages,database records) whose value is widespread. Troubleshooting systems,detecting operational trends, catching security problems and measuringbusiness performance, for example, typically require the organizationand understanding of machine data. But the overwhelming volume,different and changing formats, and overall complexity of machine datacreate substantial difficulty for software developers, systemadministrators and business people who want to make sense of it and gaininsight into information system behavior. The problem is compounded bythe fact that information systems, and the machine data they generate,continue to grow in complexity and size.

Consider for example an information system environment for web-basedapplications consisting of web servers, application servers, databasesand networks. Each information system component is constantly loggingits own machine data documenting its activities. System administratorsneed to access and comprehend the machine data from one or morecomponents to find and fix problems during operations. Security analystswant to understand patterns of machine data behavior from networkdevices to identify potential security threats. Business people areinterested in tracing the machine data across components to follow thepaths and activities customers perform when purchasing products orservices.

Today, people generally attempt to comprehend information systembehavior by manually looking at and trying to piece together machinedata using the knowledge from one or more individuals about one or moresystems. Individuals typically have specific technology domain expertiselike networking, operating systems, databases, web servers or security.This expertise can also be in specific application domains like finance,healthcare, or communications. Manual approaches can be effective whenconsidering small amounts of machine data in a single domain, but humansare easily overwhelmed as the size, variety and dynamic nature of themachine data grows.

Automated approaches, like homegrown scripts, data analysis programs,and data warehousing software, by contrast, can work with large amountsof machine data. But organizing different types of frequently changingdata and formats can be troublesome, generally requiring specificmethods for each type of data and necessitating modification of methodswhen the data formats change or new types of data are encountered.Automated approaches to building understanding from machine data aretypically limited to finding simple, predefined relationships betweenknown data elements.

Generally machine data is organized today by relying on predefined dataschemas and predetermined algorithms for parsing and categorizing data.In current approaches, what data elements exist in a machine data setand how the data elements are classified generally must be known aheadof time. How the data is cleansed, parsed and categorized is definedalgorithmically in advance for different types of data formats resultingin systems that are brittle, expensive to implement, and have numerousfunctional shortcomings. For example, unexpected types of data aretypically ignored. As a result, data categorization usefulness degradesquickly and unexpected data and behaviors are not observed or recorded.Given the inherent dynamic nature of information systems and the machinedata they generate, current organization methods have limitedapplicability.

Building understanding from machine data is inherently subjective anddepends on the task, scope of data and skill level of people using asolution. Deriving specific, useful meanings from large quantities ofmachine data can require expertise in one or more domains and knowledgeof how data from one domain relates to data from another domain. Currentmethods of deriving meaning from machine data are generally based onbuilding simple pair-wise relationships (A->B) between predetermineddata elements using data values. More advanced techniques may be able tofind predetermined multi-data element relationships (A->B->C), providedthe data elements are described in advance, requiring the availabilityof multiple domain experts to configure and continuously manage asolution.

Conventional methods, whether human or automated, of organizing andunderstanding machine data across multiple information systems anddomains suffer from an inability to effectively keep up with changingmachine data and are constrained by limited data relationships, makingthese methods difficult, time consuming, expensive and oftenineffective.

There exists, therefore, a need to develop other techniques fororganizing and deriving understanding from machine data.

SUMMARY OF THE INVENTION

Methods and apparatus consistent with the invention address these andother needs by turning machine data (MD) into a machine data web (MDW).A MDW is created by organizing MD into events representing discreteactivities, and dynamically linking events together representing larger,more complex activities. Much like the World Wide Web is a hyperlinkedinformation space of documents and web sites. A MDW is an interconnectedinformation space of information system events and activities. The MDWcan be searched, browsed, navigated, and analyzed as a proxy for theinformation-processing environment itself. Unlike the WWW's HTMLdocuments and hyperlinks, however, the events organized from machinedata, and the links between these events, do not generally exist andmust be manufactured through the processing and analysis of MD.

In one implementation, MD is organized into events using a collection oftechniques including, but not limited to, aggregating a MD collectioninto discrete events, extracting important entities from an event'sdata, segmenting an event's data into tokens, and classifying eventsinto like categories. An important aspect is the ability to continuouslylearn and adapt, keeping up with changes in the MD. In the example of aweb-based application information system environment, data sources anddata formats can be constantly changing. For example, new web serversand network components can be added and old ones removed as theapplication requires more capacity or reconfiguration.

In another aspect, knowledge or understanding is built from theorganized MD as events are connected to one another by dynamicallyconstructing links using a number of techniques, including but notlimited to the analysis of event data values, timing, patterns, andstatistics. One advantage of the MDW is that it can learn new types oflinks as they occur and build paths by chaining multiple links together.Another advantage is the ability to preserve integrity by reconstructingthe original MD from the MDW events. Dynamic construction of links andpaths through multiple machine data sources enables a systemadministrator working on a web-based application information system tofollow the sequence of activities from the web server to the applicationand eventually the database in order to locate the source of a problem.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention has other advantages and features which will be morereadily apparent from the following detailed description, when taken inconjunction with the accompanying drawings:

FIG. 1 is a diagram an example information-processing environmentsuitable for use with an MDW.

FIG. 2 is a flow diagram of one example of creation of an MDW accordingto the invention.

FIG. 3 is a flow diagram of one example of MD organization according tothe invention.

FIG. 4 is a flow diagram of one example of MD understanding according tothe invention.

FIG. 5 is a diagram illustrating access to an MOW.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the example of FIG. 1, the information-processing environmentincludes hardware and software components such as computers, routers,databases, operating systems and applications in a distributedconfiguration for processing information. Each component may beproducing MD 110, and there may be many MD sources and large quantitiesof MD across multiple technology and application domains. For example, acomputer may be logging operating system events, a router may beauditing network traffic events, a database may be cataloging databasereads and writes or schema changes, and an application may be sendingthe results of one application call to another across a message queue.In this embodiment, individual IT personnel—who may reside in differentdata centers, companies, or even geographies—typically manage specifictechnology and application domains. Aspects of the invention will bedescribed with respect to the information-processing environments inFIG. 1, but the invention can also be used with otherinformation-processing environments.

FIG. 2 represents one approach 200 to building a MDW 290 from MD 110.This approach includes an organization process 235 and an understandingprocess 275. During the organization process 235, the MD 110 isorganized into collections of discrete events 250, referred to herein asevent data (ED). Events 250 represent units of system activity. Examplesof events 250 include, for example, a web server servicing an HTTP “get”request from a web browser, an application server servicing an API call,or a database updating records in a table. Collections of events 250 candescribe larger system activities, such as an attempt to update acustomer record or submit an order. One of the challenges in organizing235 MD 110 into events 250 is that MD generally has little formalstructure and typically includes not much more than a time stamp commonacross different sources of MD and different types of events. MD 110 isalso subject to changes in environment configurations. For example,changing the verbosity level in a web server configuration file candramatically increase or decrease the amount of information included inan HTTP “get” event found in the web server's log file.

During the understanding process 275, ED 250 is analyzed to createdynamic links between events and build the MDW 290. As an example,consider that a log from a web server may contain specific types ofevents 250 with specific event data, but a log from an applicationserver or database may contain different events 250 and event dataspecific to its own domain. A system administrator may, for example,locate the web server event by looking for a session ID found in a webserver log, locate the application server event by finding a process IDin the message queue, and locate a database table update event bysearching for a transaction ID in the database audit trail. All threesources may contain events 250 that are part of a larger systemactivity, yet there is no obvious or explicit common structure or datashared among the MD 110 produced by each system. Common structure ismanufactured across the three sources by analyzing the event data 250 sothat connections between events can be identified. In oneimplementation, patterns of event behavior are recorded in real-time andidentified, for example, as frequently occurring or infrequentlyoccurring. Frequent patterns identify typical system processes andwell-known links Infrequent patterns identify deviations or anomaliesand less well-known links. Contrast this with the world of the web,where hyperlinks are part of the formal, common structure of HTML—thelanguage for building most web pages. Building links by hand for largevolumes of ED 250 is not an option for complex information-processingenvironments.

Machine Data Organization Process

FIG. 3 is a flow diagram of one implementation 235 of the MDorganization process of FIG. 2. In this implementation, there areseveral steps including collection 305, source identification 315,aggregation 325, extraction 335, segmentation 345, and classification355. Through these steps, MD 110 is collected from theinformation-processing environment and organized into ED 250 for the MDunderstanding process. For convenience, the technology that implementseach step will be referred to as a module. That is, the “collectionmodule” is the technology that collects MD. In one implementation, themodules are all implemented as software.

Collection

In the collection step 305, the MD 110 may be collected directly fromits original source or consolidated over a number of sources. Machinedata 110 can, and often does, arrive out of order. Collection 305 of MD110 can be performed based on standard approaches to data access, forexample, reading log files, examining message bus traffic, becoming async for logging systems like Syslog, or connecting to database auditingsystems. Parts of the collection module can be situated in differentlocations, preferably with access to the MD 110.

Source Identification—Classification into Domains

Given the repetitive, yet dynamic, nature of MD 110, an effectiveorganization process 235 (such as shown in FIG. 3) preferably will learnabout data formats and structure automatically. In one implementation,learning is separated into different domains based on the source of MD110. Domains can be general system types, such as log files, message bustraffic, and network management data, or specific types, such as outputof a given application or technology—Sendmail logging data, Oracledatabase audit data, and J2EE messaging. An MDW can include a mix ofgeneral domains and specific domains.

In this example organization process 235, the domain for a given sourceof MD is identified 315 so that domain specific organization methods canbe applied. Domains are determined through a learning process. Thelearning process uses collections of MD from well-known domains as inputand creates a source signature 312 for each domain. In oneimplementation, source signatures 312 are generated from representativesamples of MD 110 by creating a hash table mapping punctuationcharacters to their frequency. While the tokens and token values canchange in a MD collection, in this particular implementation, thesignature 312 generated by the frequency of punctuation is quite stable,and reliable within a specific domain. Other implementations could usefunctions of the punctuation and tokens, such as the frequencies of thefirst punctuation character on a line, or the first capitalized term ona line. Given that source signatures 312 can be large and hard to read,signatures can have a corresponding label in the form of a number ortext that can be machine generated or human assigned. For example, thesource signature 312 for an Apache web server log might beprogrammatically assigned the label “205”, or a user can assign thelabel “Apache Server Log”.

In one embodiment, clustering is used to classify 315 collected MD 110into domains according to their source signatures 312. As collections ofMD 110 are encountered, each collection's signature is matched to theset of known source signatures 312 by performing a nearest-neighborsearch. If the distance of the closest matching signature 312 is withina threshold, the closest matching signature 320's domain is assumed tobe the domain of the source. If no best match can be found, a new sourcesignature 312 can be created from the sample signature and a new sourcedomain created. Alternatively, a default source domain can be used. Inone implementation, the distance between two signatures is calculated byiterating over the union of attributes of the two signatures, with thetotal signature distance being the average of distances for eachattribute. For each attribute A, the value of A on Signature1 andSignature2, V1 and V2, are compared and a distance is calculated. Thedistance for attribute A is the square of (V1−V2)*IDF, where IDF is thelog(N/|A|), where N is the number of signatures, and |A| is the numberof signatures with attribute A.

Source Identification—Classification as Text/Binary

Some MD 110 sources are non-textual or binary and cannot be easilyprocessed unless a known process is available to convert the binary MDinto textual form. To classify a source as textual or binary, a sampleMD collection is analyzed. Textual MD can also have embedded binary MD,such as a memory dump, and the classification preferably identifies itas such. In one implementation, the textual/binary classification worksas follows. The sample is a set of lines of data, where a line isdefined as the data between new lines (i.e., ‘\n’), carriage-returns(i.e., ‘\r’), or their combination (i.e., ‘\r\n’). For each line, if theline's length is larger than some large threshold, such as 2kcharacters, or if the line contains a character with an ASCII value ofzero (0), a count of Binary-looking lines is incremented. Otherwise, ifthe line's length is shorter than a length that one would expect mosttext lines to be below, such as 256 characters, a count of Text-lookinglines is incremented. If the number of Text-looking lines is twice asnumerous as the Binary-looking lines (other ratios can be used dependingon the context), the source is classified as text. Otherwise, the sourceis classified as binary.

Aggregation of Machine Data into Raw Events

When the source signature 320 for a collection of MD has been identified315, the corresponding aggregation rules are applied 325 to the MDcollection. Aggregation rules describe the manner in which MD 110, froma particular domain, is organized 325 into event data 330 by identifyingthe boundaries of events within a collection of MD, for example, how tolocate a discrete event by finding its beginning and ending. In oneimplementation, the method of aggregation 325 learns, without priorknowledge, by grouping together multiple lines from a sample of MD 110.Often MD 110 contains events 330 that are anywhere from one to hundredsof lines long that are somehow logically grouped together.

The MD collection may be known a priori, or may be classified, assingle-line type (i.e., containing only single-line events) ormulti-line type (i.e., possibly containing multi-line events) prior toperforming aggregation. For those MD collections that are classified assingle-line type, aggregation 325 is simple—single-line type MDcollections are broken on each line as a separate event. Multi-line typeMD collections are processed 325 for aggregation. In one implementation,a MD collection is classified as a multi-line type if 1) there is alarge percentage of lines that start with spaces or are blank (e.g., ifmore than 5% of the lines start with spaces or are blank), or 2) thereare too many varieties of punctuation characters in the first Npunctuation characters. For example, if the set of the first threepunctuation characters found on each line has more than five patterns(e.g., ‘:::’, ‘!:!’, ‘,,,’, ‘:,,’, ‘( )*’), the collection might beclassified as multi-line.

Another aspect of aggregation methods 325 is the ability to learn, andcodify into rules, what constitutes a break between lines and thereforethe boundary between events, by analyzing a sample of MD. For example,in one implementation, an aggregation method 325 compares every two-linepair looking for statistically similar structures (e.g., use of whitespace, indentation, and time-stamps) to quickly learn which two belongtogether and which two are independent. In one implementation,aggregation 325 works as follows. For each line, first check if the linestarts with a time-stamp. If so, then break. Typically, lines startingwith a time-stamp are the start of a new event. For lines that do notstart with a time-stamp, combine the current line with the prior line tosee how often the pair of lines occurs, one before the other, as apercentage of total pairs in the MD sample. Line signatures are used inplace of lines, where a line signature is a more stable version of aline, immune to simple numeric and textual changes. In thisimplementation, signatures can be created by converting a line into astring that is the concatenation of leading white space, any punctuationon the line, and the first word on the line. The line “10:29:03 Host191.168.0.1 rebooting:normally” is converted to “::..:Host.”

Now this current line signature can be concatenated with the previousline signature (i.e., signature1 combined with signature2) and used as acombined key into a table of break rules. The break rule table maps thecombined key to a break rule, which determines whether there should be a‘break’, or not, between the two lines (i.e., whether they are part ofdifferent events or not). Break rules can have confidence levels, and amore confident rule can override a less confident rule. Break rules canbe created automatically by analyzing the co-occurrence data of the twolines and what percent of the time their signatures occur adjacently. Ifthe two line signatures highly co-occur, a new rule would recommend nobreak between them. Alternatively, if they rarely co-occur, a new rulewould recommend a break between them. For example, if line signature Ais followed by line signature B greater than 20% of the time A is seen,then a break rule might be created to recommend no break between them.Rules can also be created based on the raw number of line signaturesthat follow/proceed another line signature. For example, if a linesignature is followed by say, ten different line signatures, create arule that recommends a break between them. If there is no break rule inthe break rule table, the default behavior is to break and assume thetwo lines are from different events. Processing proceeds by processingeach two-line pair, updating line signature and co-occurrencestatistics, and applying and learning corresponding break rules. Atregular intervals, the break rule table is written out to the hard diskor permanent storage.

Extraction of Entities

Following aggregation 325 and before event segmentation 345, variousextraction methods 335 can be applied to identify semantic entities 340within the data. In one implementation, search trees or regularexpressions can be applied to extract and validate, for example, IPaddresses or email addresses. The goal of extraction 335 is to assistthe segmentation process 345 and provide semantic value to the data.

Segmentation of Events

Segmentation 345 rules describe how to divide event data 330 intosegments (also known as tokens 350). It is important to note at thispoint that segments 350 have little semantic value, unless an extractedentity 340 has been applied. In one implementation a segmentation rule345 examines possible separators or punctuation within the event 330,for example, commas, spaces or semicolons. An important aspect ofsegmentation 345 is the ability to not only identify individual segments350, but also to identify overlapping segments 350. For example, thetext of an email address, “bob.smith@corp.com”, can be broken 345 intoindividual and overlapping segments 350; <bob.smith>, <@> and <corp.com>can be identified as individual segments, and <<bob.smith><@><corp,com>>can also be identified as an overlapping segment. In one implementation,segmentation 345 uses a two-tier system of major and minor breaks. Majorbreaks are separators or punctuation that bound the outer most segment350. Examples include spaces, tabs, and new lines. Minor breaks areseparators or punctuation that break larger segments 350 into subsegments 350, for example periods, commas, and equal signs. In oneimplementation, more complex separators and punctuation combinations areused to handle complex segmentation tasks 345, for example handling Javaexceptions in an application server log file.

Classification of Event Types

In the embodiment of FIG. 3, the final step of the organization process235 is the classification 355 of events 350 into event types. Examplesof event types include a web server HTTP “get,” an application serverdatabase “connect,” or an email server “send mail attempt.” In oneimplementation, an event signature 352 is generated for each event type.One method for generating an event signature 352 is to build ahierarchical scheme for identifying particular types of events based onthe overall event structure 330, segmentation 350, segment values 350,and extracted entities 340. The purpose of the event signature 352 is toidentify a type of event regardless of the situation. In this way aparticular type of event can have the same signature 352 in multipleMDWs. For example, a mail server's send mail attempt generally has thesame signature 352 in every MDW regardless of the information-processingenvironment.

In one implementation a hierarchical event signature {v1, v2, v3, . . .vn} 352 is constructed from a list of successively more specific hashfunctions {f1( ), f2( ), f3( ), . . . fn( )}, where each fn( ) producesa value representing a level of the hierarchy. The event signature 352is most useful when each successive function is more specific. Forexample, in one embodiment, the following function list represents a 9level event signature 352, from most general to most specific:

-   -   f1( ): firstCharType—returns alpha, numeric, white space, other,        depending on the type of the first character of the event.    -   f2( ): headwhitespace—returns the number of spaces/tabs at the        beginning of the event.    -   f3( ): firstpunc—returns the first punctuation character of the        event.    -   f4( ): firstImportantKeywords—returns a hash value of first word        in the event that is an important keyword, where there is a list        of known important terms.    -   f5( ): firstKnownWord—returns the first word in the event that        is a known keyword, where there is a list of known terms.    -   f6( ): importantKeywords returns the list of all hash values of        important keywords that are found in the event.    -   f7( ): firstUnknownWord—returns the first word in event that is        not a known keyword.    -   f8( ): headPunc—returns the first 10 punctuation characters in        the event, removing duplicates.    -   f9( ): allPunc—returns all punctuation in event.

In this implementation, the event signature 352 is a traversal through ahierarchy of possible values. Given that event signatures 352 can belarge and hard to read, an event signature can have a correspondinglabel in the form of a number or text that can be machine generated orhuman assigned. For example, an email server “send mail attempt” eventmight be programmatically assigned the label “500”, but a user canassign the label “send mail attempt”.

Machine Data Understanding Process

FIG. 4 is a flow diagram of one implementation 275 of the MDunderstanding process shown in FIG. 2. During the understanding process275, knowledge about how events relate to one another is discovered fromthe event data 250. This knowledge is valuable in understanding thebehavior of the underlying information-processing environment. Links410, representing relationships between events 250 are useful, amongother things, for finding connections and causality where little or nocommon structure exists. For example, in an email-messaginginformation-processing environment, an event 250 may exist in themessage transfer agent (MTA) indicating the receipt of a message from asender, another event 250 may exist in the spam filtering softwaredocumenting that the sender is known and the message is safe to forwardto a user's mailbox, and finally the mailbox authentication may containan event 250 showing that the user attempted to login to their mailboxand retrieve their mail. These three events 250 may contain no commonstructure other than a timestamp. However, the three events 250 areconnected as part of a larger email messaging activity. In oneimplementation of the understanding process 275, several techniques areapplied including linking 405, which creates connections 410 betweenevents 250; path construction 415, to build more complex, multi-linkconnections 420; and analysis 425, which records historical data 492 andgenerates statistics 494 about the MDW.

Linking Events

By analyzing event data 250 and possible link hints 402 from externalsystems or human input, links 410 can be created 405. An importantfeature of the MDW approach is the ability to create 405 linkrelationships 410 dynamically and learn new possible link relationshipson the fly. A number of methods can be used in the analysis of ED 250 tocreate 405 links 410, including, but not limited to, value analysis,statistical analysis, timing analysis, and the evaluation of link hints402. These methods can be used individually or in combination with oneanother. From our previous example, perhaps the link 410 between the MTAand the spam filter events 250 is a value association between the MTAmessage ID and the spam filter article ID, or the link 410 between thespam filter and the user email retrieval 250 is an associative mail boxname, All three events 250 might be tied together, for example byobserving a timing pattern that occurs over and over again withstatistically relevant frequency.

In one implementation, link analysis 405 takes place by creating aco-occurrence table with an entry for pairs of event types or event datavalues that occur within a predetermined window of each other. In oneaspect, windows are bounded by a window threshold taking the form oftime (e.g. 10 minutes), event types (e.g. 50 unique event types), orevent instances (e.g. 1000 events). The value of the co-occurrence tableentry is the distance between the pair (time, event types, or eventinstances). Pairs that co-occur often enough, and meet a distancestandard deviation threshold are deemed relevant and reliable links. Forexample, assume that an event 250 of type A occurred 50 times, an eventof type B occurred 40 times, an event of type A was followed by an eventof type B 20% of the time, and the standard deviation of their distancewas less than 5.0 (a predetermined threshold), then a link 410 iscreated between events 250 of type A and type B (represented as A->B).Standard deviation thresholds are based on a function of windowthresholds and may change based on the time to complete analysis or thenumber of desired results. Window thresholds may change based on datadensity and time available to complete the analysis.

Path Construction by Chaining Linked Events

Paths 420 are multi-link collections representing a chain of linkedevents 410. Paths 420 often represent a higher level of informationsystem behavior, possibly spanning multiple systems, applications ordata centers. Paths 420 are useful, for example, for following morecomplex activities or transactions through one or more systems. In ouremail example, a path 420 could be the receiving or sending of an emailincluding three or more events 250 and two or more links 410. Similar tolinks 410, paths 420 are created 415 by analyzing event data 250,410 andpossible path hints 412 from external systems or human input. Animportant feature is the ability to create paths 420 dynamically andlearn new possible paths on the fly.

Paths 420 are built by chaining together 415 event links 410, using anumber of methods, In one implementation, paths 420 are discovered aschains of transitive links 410. For example, given previously discoveredlinks 410 A->B, B->C, A->C, and C->A, transitively composition yieldsthe following three event paths 420: A->B->C, B->C->A, A->C->A, C->A->Band C->A->C. These paths 420 can also be combined to make larger andlarger path chains. In one aspect, certain restrictions are applied 415to reduce combinatorial explosion. One restriction might involve theelimination of cycles and repetitions. For example, one rule 415 mightbe that A->C and C->A cannot be combined to create A->C->A. In a secondpossible restriction 415, for A->B and B->C to be combined there must bean A->C link 410, with the average distance of A->C being approximatelyequal to the sum of the average distances between A->B and B->C. Inaddition, the standard deviation of the distance for A->C must beapproximately equal to the standard deviations of A->B and B->C.Finally, paths 420 that are rotations of other paths can be removed,keeping the most reliable path. For example, given paths 420 A->B->C andC->A->B, if the standard deviation of the distance between C->A isgreater than the standard deviation of the distance between B->C thenA->B->C would be kept and C->A->B removed.

Like the WWW and HTML hyperlinks, event links 410 and paths 420 can berepresented as a uniform resource locator (URL). In one implementation alink 410 from one event 250 to another is represented by the followingURL “mdw://<name of MDW>/<link type>/<link value>/<event 1>/event <2>.”A link 410 can resolve to one of several destinations including, but notlimited to an event type, an event instance or an event segment withinan event instance.

Analysis of the MDW

In addition to links 410 and paths 420, another aspect of the MDWunderstanding process 275 is the ability to generate 425 historicalinformation 492 about itself, for example, statistics 494 for event,event type, link or path occurrences. One aspect of historical data 492regarding the MDW is that it can reveal historical behavior of theinformation-processing environment itself.

Accessing the MDW

FIG. 5 refers to one approach to access the elements of the machine dataweb 290, including its data and dynamic relationships, through anapplication-programming interface (API). In one embodiment, the MDW 290and corresponding technology infrastructure is Internet-based. The APIincludes commands to post data 510 to the MDW infrastructure 290including, but not limited to, MD, events, segments, source signatures,link hints, and path hints. In the same embodiment, the API alsoincludes commands to get data 520 from the MDW 290 including, but notlimited to, the original MD, events, segments, source signatures, links,and paths. Utilizing the MDW API, a variety of applications and systemscan take advantage of an advanced organization and understanding ofmachine data.

The MDW can be implemented in many different ways. In one approach, eachbox in FIGS. 2, 3 and 4 is implemented in software as a separateprocess. All of the processes can run on a single machine or they can bedivided up to run on separate logical or physical machines. In alternateembodiments, the invention is implemented in computer hardware,firmware, software, and/or combinations thereof. Apparatus of theinvention can be implemented in a computer program product tangiblyembodied in a machine-readable storage device for execution by aprogrammable processor. Method steps of the invention can be performedby a programmable processor executing a program of instructions toperform functions of the invention by operating on input data andgenerating output. The invention can be implemented advantageously inone or more computer programs. Each computer program can be implementedin a high-level procedural or object-oriented programming language or inassembly or machine language if desired; in any case, the language canbe a compiled or interpreted language. Any of the foregoing can besupplemented by, or incorporated in, ASICs (application-specificintegrated circuits) and other forms of hardware.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. For example, not all of thesteps shown are required in every implementation, and they may beimplemented in ways other than the examples given above. The order ofthe steps may also be changed in certain cases. Various othermodifications, changes and variations which will be apparent to thoseskilled in the art may be made in the arrangement, operation and detailsof the method and apparatus of the present invention disclosed hereinwithout departing from the spirit and scope of the invention.

What is claimed is:
 1. A method, comprising: receiving machine data fromtwo or more components in an information technology environment, themachine data reflecting activity in the information technologyenvironment; identifying behavioral patterns among the machine data fromthe two or more components in the information technology environment;analyzing the identified behavioral patterns to determine one or morebehavioral patterns of the identified behavioral patterns thatfrequently occur; designating the one or more behavioral patterns thatfrequently occur as representing typical system behavior in theinformation technology environment; wherein the method is performed byone or more computing devices.
 2. The method as recited in claim 1,wherein the typical system behavior identifies typical system processes.3. The method as recited in claim 1, wherein the typical system behavioridentifies relationships between events that co-occur in the machinedata.
 4. The method as recited in claim 1, wherein the typical systembehavior identifies relationships between events that co-occur in themachine data within a particular time window.
 5. The method as recitedin claim 1, wherein the typical system behavior identifies events thatoccur in at least a portion of the two or more components that areconnected as part of a larger activity in the information technologyenvironment.
 6. The method as recited in claim 1, wherein machine dataproduced from a first component of the two or more components has adifferent data format than machine data produced from a second datacomponent of the two or more components.
 7. The method as recited inclaim 1, wherein the analyzing the identified behavioral patternsfurther comprises: identifying whether a number of occurrences of aparticular behavioral pattern exceeds a threshold.
 8. The method asrecited in claim 1, wherein the identifying behavioral patterns furthercomprises: analyzing the machine data from the two or more components toidentify a relationship between portions of the machine data.
 9. Themethod as recited in claim 1, wherein the identifying behavioralpatterns further comprises: analyzing machine data from the two or morecomponents to identify a relationship between portions of the machinedata; linking the related portions of machine data to each other. 10.The method as recited in claim 1, further comprising: analyzing themachine data in order to segment the machine data into a plurality ofevents by determining a beginning and ending of each event in theplurality of events in the machine data, each event in the plurality ofevents including a portion of the machine data segmented for that event.11. The method as recited in claim 1, further comprising: analyzing themachine data in order to segment the machine data into a plurality ofevents by determining a beginning and ending of each event in theplurality of events in the machine data, each event in the plurality ofevents including a portion of the machine data segmented for that event;associating a time stamp with each event in the plurality of events, thetime stamp derived from machine data included in that event.
 12. One ormore non-transitory computer-readable storage media, storing one or moresequences of instructions, which when executed by one or more processorscause performance of: receiving machine data from two or more componentsin an information technology environment, the machine data reflectingactivity in the information technology environment; identifyingbehavioral patterns among the machine data from the two or morecomponents in the information technology environment; analyzing theidentified behavioral patterns to determine one or more behavioralpatterns of the identified behavioral patterns that frequently occur;designating the one or more behavioral patterns that frequently occur asrepresenting typical system behavior in the information technologyenvironment.
 13. The one or more non-transitory computer-readablestorage media as recited in claim 12, wherein the typical systembehavior identifies typical system processes.
 14. The one or morenon-transitory computer-readable storage media as recited in claim 12,wherein the typical system behavior identifies relationships betweenevents that co-occur in the machine data.
 15. The one or morenon-transitory computer-readable storage media as recited in claim 12,wherein the typical system behavior identifies relationships betweenevents that co-occur in the machine data within a particular timewindow.
 16. The one or more non-transitory computer-readable storagemedia as recited in claim 12, wherein the typical system behavioridentifies events that occur in at least a portion of the two or morecomponents that are connected as part of a larger activity in theinformation technology environment.
 17. The one or more non-transitorycomputer-readable storage media as recited in claim 12, wherein theanalyzing the identified behavioral patterns further comprises:identifying whether a number of occurrences of a particular behavioralpattern exceeds a threshold.
 18. The one or more non-transitorycomputer-readable storage media as recited in claim 12, wherein theidentifying behavioral patterns further comprises: analyzing the machinedata from the two or more components to identify a relationship betweenportions of the machine data.
 19. The one or more non-transitorycomputer-readable storage media as recited in claim 12, furthercomprising: analyzing the machine data in order to segment the machinedata into a plurality of events by determining a beginning and ending ofeach event in the plurality of events in the machine data, each event inthe plurality of events including a portion of the machine datasegmented for that event; associating a time stamp with each event inthe plurality of events, the time stamp derived from machine dataincluded in that event.
 20. An apparatus, comprising: one or moreprocessors; and a memory storing instructions, which when executed bythe one or more processors, causes the one or more processors to:receive machine data from two or more components in an informationtechnology environment, the machine data reflecting activity in theinformation technology environment; identify behavioral patterns amongthe machine data from the two or more components in the informationtechnology environment; analyze the identified behavioral patterns todetermine one or more behavioral patterns of the identified behavioralpatterns that frequently occur; designate the one or more behavioralpatterns that frequently occur as representing typical system behaviorin the information technology environment.